Author: sanjayshonak_wxo2cg

DB Admin: Database size in TBs

Posted on by 
SET SERVEROUTPUT ON
declare
total_size_b number;
tfree_size_b number;
tused_size_b number;
begin
dbms_output.enable(100000);
select
Sum(bytes) into total_size_b
from
dba_data_files;
select
Sum(bytes) into tfree_size_b
from
dba_free_space;
select
Sum(bytes) into tused_size_b
from
dba_segments;
dbms_output.put_line('Total:' || TO_CHAR(Round(total_size_b/1024/1024/1024/1024, 2), '999,999.00') || ' TB');
dbms_output.put_line('Free: ' || TO_CHAR(Round(tfree_size_b/1024/1024/1024/1024, 2), '999,999.00') || ' TB');
dbms_output.put_line('Used: ' || TO_CHAR(Round(tused_size_b/1024/1024/1024/1024, 2), '999,999.00') || ' TB');
end;
/
SET SERVEROUTPUT OFF

Database: Oracle: Tablesapces size(MB), free(MB) % used and %free

Posted on by 
 set pagesize 150
SELECT /* + RULE */ df.tablespace_name "Tablespace",
df.bytes / (1024 * 1024) "Size (MB)",
SUM(fs.bytes) / (1024 * 1024) "Free (MB)",
Nvl(Round(SUM(fs.bytes) * 100 / df.bytes),1) "% Free",
Round((df.bytes - SUM(fs.bytes)) * 100 / df.bytes) "% Used"
FROM dba_free_space fs,
(SELECT tablespace_name,SUM(bytes) bytes
FROM dba_data_files
GROUP BY tablespace_name) df
WHERE fs.tablespace_name (+) = df.tablespace_name
GROUP BY df.tablespace_name,df.bytes
UNION ALL
SELECT /* + RULE */ df.tablespace_name tspace,
fs.bytes / (1024 * 1024),
SUM(df.bytes_free) / (1024 * 1024),
Nvl(Round((SUM(fs.bytes) - df.bytes_used) * 100 / fs.bytes), 1),
Round((SUM(fs.bytes) - df.bytes_free) * 100 / fs.bytes)
FROM dba_temp_files fs,
(SELECT tablespace_name,bytes_free,bytes_used
FROM v$temp_space_header
GROUP BY tablespace_name,bytes_free,bytes_used) df
WHERE fs.tablespace_name (+) = df.tablespace_name
GROUP BY df.tablespace_name,fs.bytes,df.bytes_free,df.bytes_used
ORDER BY 4 DESC;

Note: Least free % tablespace will 1st in output. Also it include temp tablespace/s.


AWS commands in a nutshell

Posted on by

AWS CLI is an common CLI tool for managing the AWS resources. With this single tool we can manage all the aws resources.

sudo apt-get install -y python-dev python-pip 
sudo pip install awscli 
aws --version 
aws configure

Bash one-liners

cat # output a file
tee # split output into a file
cut -f 2 # print the 2nd column, per line
sed -n ‘5{p;q}’ # print the 5th line in a file
sed 1d # print all lines, except the first
tail -n +2 # print all lines, starting on the 2nd
head -n 5 # print the first 5 lines
tail -n 5 # print the last 5 lines

expand # convert tabs to 4 spaces
unexpand -a # convert 4 spaces to tabs
wc # word count
tr ‘ ‘ \t # translate / convert characters to other characters

sort # sort data
uniq # show only unique entries
paste # combine rows of text, by line
join # combine rows of text, by initial column value

Cloudtrail – Logging and Auditing

list all trails
 aws cloudtrail describe-trails
 list all S3 buckets
 aws s3 ls
 create a new trail
 aws cloudtrail create-subscription \
     --name awslog \
     --s3-new-bucket awslog2016
 list the names of all trails
 aws cloudtrail describe-trails --output text | cut -f 8
 get the status of a trail
 aws cloudtrail get-trail-status \
     --name awslog
 delete a trail
 aws cloudtrail delete-trail \
     --name awslog
 delete the S3 bucket of a trail
 aws s3 rb s3://awslog2016 --force
 add tags to a trail, up to 10 tags
 aws cloudtrail add-tags \
     --resource-id awslog \
     --tags-list "Key=log-type,Value=all"
 list the tags of a trail
 aws cloudtrail list-tags \
     --resource-id-list 
 remove a tag from a trail
 aws cloudtrail remove-tags \
     --resource-id awslog \
     --tags-list "Key=log-type,Value=all"

Amazon Simple Email Service (SES)

Posted on by

In this article we will see how to use the SES.

Once logged in aws console click the ‘Simple Email Service’ under the ‘Customer Engagement’. You will see below option.

‘Simple Email Service‘ under the ‘Customer Engagement’.
Using any of the above options you can perform as relevant task.
There are multiple options under the ‘SES’ home (left side panel).
Under ‘SMTP Settings’ click the highlighted button ‘Create My SMTP Credentials’

Clock ‘Create’
Once you will Create Record Sets an email will be sent
Once you click on ‘Use Route 53’ (background page below ‘Create Record Sets).

Create Receipt Rule

Click ‘Go to rule set creation’ (below last icon).
Create a Receipt Rule.
Click ‘Add Recipient.
Note: Once clicked you will see the receipt add and ‘Verification Status’ and option to remove.

AWS S3(Simple Storage Service and buckets

Posted on by

S3 (Simple Storage Service)

How to access the S3?

Go to “Services” ==> S3, please see the below screenshot for more details.

Click the cross top left and you will be able to see the options:

This image has an empty alt attribute; its file name is image-15-1024x423.png
Bucket options: Create, Delete, Empty and edit public access etc.
Click ‘Edit’ on top right and you will see the options with check boxes, next scren
You can manage ‘Public Access List (ACLs) and ‘Public Bucket Policies’ for account.
  1. S3 has a simple web services interface that you can use to store and retrieve any amount of data, at any time, from anywhere on the web. 
  2. It gives any developer access to the highly scalable, reliable, fast, inexpensive data storage infrastructure.
  3. S3 Intelligent-Tiering:  You no longer need to think about which storage class to store data in to optimize storage costs. The S3 Intelligent-Tiering storage class automatically moves your data to the most cost-effective storage access tier. No more custom policies or code needed. It is the ideal storage class for data with unknown or changing access patterns.
  4. S3 Block public access: You can prevent public access to any bucket or object with just a few clicks on the S3 console.Use S3 Block public access to prevent public access to your existing and new buckets and objects. You can block public access at the account level and at the bucket level. Block public access settings are easy to audit. You can also configure them using the AWS CLI, AWS SDKs, the S3 REST APIs, and from within AWS CloudFormation templates.
  5. S3 Batch operations: Now you can apply a change, like replacing an access control list (ACL), to millions (and billions) of objects without writing a custom application. Use S3 Batch operations to specify a group of objects (a bucket or specific objects defined in a custom manifest or an S3 inventory report) and an action to take on those objects. The actions include replace object ACL, initiate a restore from S3 Glacier, copy objects, and more.
  6. S3 Glacier is a really low cost storage service that provide secure, durable and flexible storage for backup and archival data.

How S3 batch operation works?

S3 Glacier

S3 Glacier is a really low cost storage service that provide secure, durable and flexible storage for backup and archival data.

You can create ‘Vault’ and set retrieval policies and event notifications.

Amazon EBS

Posted on by
  1. Elastic Block Store (EBS) provides persistent block storage volumes for use with EC2 instances in the AWS Cloud.
  2. Each EBS volume is automatically replicated within its Availability Zone to protect you from component failure, offering high availability and durability. 
  3. EBS volumes offer the consistent and low-latency performance needed to run your workloads.
  4. With Amazon EBS, you can scale your usage up or down within minutes.
  5. EBS volumes are particularly well-suited for use as the primary storage for file systems, databases, or for any applications that require fine granular updates and access to raw, unformatted, block-level storage.
  6. EBS is well suited to both a. database-style applications that rely on random reads and writes. b. applications that perform long, continuous reads and writes.

AWS Basic architecture

Posted on by

Note: Don’t confuse EC2 with S3.

Because S3 is a repository for Internet data which provides access to reliable, fast, and inexpensive data storage infrastructure. S3 is designed to make web-scale computing easy by enabling you to store and retrieve any amount of data, at any time, from within Amazon EC2  OR anywhere on the web.

EC2 Instance:(Elastic Compute Cloud)

  1. Elastic Compute Cloud EC2 instance is like a remote computer running Windows or Linux and on which you can install whatever software you want, including a Web server running PHP code and a database server.
  2. EC2 is an Infrastructure as a Service (IaaS) Cloud Computing Platform provided by Amazon Web Services, that allows users to instantiate various types of virtual machines.
  3. EC2 provides scalable computing capacity in the Amazon Web Services (AWS) cloud. Using Amazon EC2 eliminates your need to invest in hardware up front, so you can develop and deploy applications faster.
  4. EC2 FAQs
  5. More details about EC2

EBS: (Elastic Block Storage) (EBS store data using buckets)

  1. EBS stands for Elastic Block Storage, and is a service that provides dynamically allocatable, persistent, block storage volumes that can be attached to EC2 instances.
  2. Most system operations that can be performed with an HDD can be performed with an EBS volume. e.g. – formatted with a filesystem and mounted.
  3. EBS also provides additional SAN-like features such as taking snapshots of volumes, and detaching and reattaching volumes dynamically.
  4. One notable feature that SAN LUNs support that EBS volumes do not is muti-initiator. (IE: Only a single EC2instance can be associated with a given EBS volume at a given time, so shared storage clustering is currently not supported.).
  5. EBS FAQs
  6. More about EBS

S3 (Simple Storage Service)

  1. Amazon S3 has a simple web services interface that you can use to store and retrieve any amount of data, at any time, from anywhere on the web.
  2. It gives any developer access to the same highly scalable, reliable, fast, inexpensive data storage infrastructure that Amazon uses to run its own global network of web sites.
  3. How to use an S3 bucket? a. First creates a bucket in the AWS region of his or her choice and gives it a globally unique name. AWS recommends that customers choose regions geographically close to them to reduce latency and costs. b. Once the bucket has been created, the user then selects a tier for the data. c. Bucket name are unique across aws.
  4. An AWS customer can interact with an Amazon S3 bucket using any of 3 methods. a. AWS Management Console. b.AWS Command Line Interface. c. application programming interfaces (APIs).
  5. S3 FAQs
  6. Really good article about “What S3 is not”?
  7. More about S3 and buckets
  8. NOTES: There are three tiers of S3 Storage available: S3 Standard – Durable, immediately available suitable for frequently accessed data. By default, data stored in S3 is written across multiple devices in multiple locations providing resiliency. (SLA: 99.99% availability & 99.99999999999% durability). S3 IA (Infrequently Accessed) – This is the same service as S3 although available at a lower cost. S3 IA users pay a retrieval fee meaning it is only a cost effective storage option for data that isn’t frequently accessed. Reduced Redundancy Storage – A lower cost storage solution with reduced SLAs (SLA: 99.99% availability & durability). c. Then, the user can specify access privileges for the objects stored in a bucket, through mechanisms such as the AWS Identity and Access Management service, bucket policies and access control lists.

FAQs on Amazon Elastic Compute Cloud (EC2).

Posted on by

Q: What is EC2?

A: EC2 is an Infrastructure as a Service Cloud Computing Platform provided by Amazon Web Services, that allows users to instantiate various types of virtual machines.

Q: What is an instance?

A: An EC2 instance is a Virtual Machine running on Amazon’s EC2 Cloud.

Q: What is an AMI?

A: An AMI (Amazon Machine Image) is a preconfigured bootable machine image, that allows one to instantiate an EC2 instance. (EC2 Virtual Machine)

Q: What is an AKI?

A: An AKI (Amazon Kernel Image) is a preconfigured bootable kernel miniimage, that are prebuild and provided by Amazon to boot instances. Typically one will use an AKI that contains pv-grub so that one can instantiate an instance from an AMI that contains it’s own Xen DomU kernel that is managed by the user.

Q: What is EBS?

  1. EBS stands for Elastic Block Storage, and is a service that provides dynamically allocatable, persistent, block storage volumes that can be attached to EC2 instances.
  2. Most system operations that can be performed with an HDD can be performed with an EBS volume. e.g. – formatted with a filesystem and mounted.
  3. EBS also provides additional SAN-like features such as taking snapshots of volumes, and detaching and reattaching volumes dynamically.
  4. One notable feature that SAN LUNs support that EBS volumes do not is muti-initiator. (IE: Only a single EC2instance can be associated with a given EBS volume at a given time, so shared storage clustering is currently not supported.)

Q: What is the difference between an instance-store AMI/instance and an EBS AMI/instance?

Answer:

  1. An instance-store instance boots off of an AMI that instantiates a non-persistent root volume that loses all data on poweroff, or hardware failure.
  2. EBS instances boot off an AMI that consists of an EBS volume that persists after powering off (stopping) an instance or in the event of a hardware failure a given instance is running on. EBS root volumes can be snap-shotted and cloned, like other EBS volumes.

Q: What is the difference between terminating an instance and stopping an instance?

A: Please note this difference is only applicable to EBS-root instances.

  1. When one stops an instance it basically virtually powers off the instance but it remains in the inventory to be powered on (started) again.
  2. Terminating an instance removes its records from the system inventory and usually also deletes its root volume.

Q: How does IP addressing work in EC2?

A: Modern EC2 instances typically exist in a “VPC”, or Virtual Private Cloud network. A VPC is a network overlay environment allowing the user to specify various aspects of the network topology including CIDR ranges, subnets, routing tables, and ACLs. Instances are assigned one or more network interfaces in a VPC, each of which can have one or more IP addresses. Publicly routable IPv6 addresses are available. IPv4 addressing is handled using private RFC 1918 addresses and stateless 1:1 NAT for public internet access.

A legacy “classic” networking mode exists in which each instance is given a randomly assigned private IP address that maps via NAT to an also randomly assigned public IP address. Amazon is not provisioning this feature for new accounts. VPC instances allow more control of the private (and public) IP address mappings and assignment, and as such let one assign custom private IP ranges and addresses, in addition to having the option to not assign public IP address mappings.

Q: What is an Elastic IP Address (EIP)?

A: An Elastic IP address is a Public IP addressed that is assigned to an individual AWS account. These IPs are assigned by region. This address can be assigned to any EC2 instance within a region and will replace the regularly assigned random public IP address.

Q: What is an EC2 Region?

A: An EC2 Region refers to a geographic region that is a completely autonomous set of compute resources, with their own management infrastructure. Regions do not share any resources, so they are considered completely separate for disaster recovery purposes.

Q: What EC2 Regions are there?

A: The official list of regions grows with some regularity. In general, the latest Debian AMIs are available in all public regions. There is also the non-public GovCloud region, available only to US Government agencies. At present, Debian AMIs are not published in GovCloud, but users have requested them.

Q: What is an EC2 Availability Zone?

A: An availability zone is a separate “failure zone” within a given region that can have resources instantiated in. Each region has it’s own power grid, and physical set of hardware and resources. Availability zones within a given region have a shared management interface/infrastructure.

Q: What is an EC2 Security Group?

A: A Security Group (SG) is a management construct within EC2 that acts similarily to a network based firewall. An instance must be assigned one or more security groups at first instantiation. Security group membership may not change after initial instantiation. Security groups allow one to set incoming network rules allowing certain TCP/UDP/ICMP protocols ingress via rules based on incoming security group ID, network address or IP address. Security groups do not restrict outbound traffic, nor do they restrict traffic between instances within the same security group. (Assuming they are communicating via their private IP addresses.)

Q: What is instance metadata?

A: Instance metadata is descriptive information about a particular instance, that is available via an http call to a particular instance and that instance alone. e.g. – Public IP address, availability zone, etc. userdata is one of these pieces of data available.

Q: What is userdata?

A: When one instantiates an EC2 instances one may optionally pass 16 KB of data to the API that can be used by the instance. (Typically use cases are running scripts, and/or configuring the instance to meet a particular use case.)

Q: What is cloud-init?

A: Cloud-init is a framework written in Python for handling EC2 userdata to configure a newly instantiated EC2 instance. See upstream project for more details: https://help.ubuntu.com/community/CloudInit

Q: How do I log into a Debian EC2 instance for the first time?

A: When you instantiate an instance from an official Debian AMI, one needs to assign a previously uploaded/created ssh public key, which will be added to the “admin” user’s authorized_keys. One can then ssh in as “admin” and use sudo to add additional users.

Q: What are the different methods supported to manage EC2?

A: Either via the AWS Web Console, via the API, or via CLI tools.

Q: How do I get to the AWS Web Console?

A: https://console.aws.amazon.com/

Q: Where is the EC2 API documented?

A: http://docs.amazonwebservices.com/AWSEC2/latest/APIReference/Welcome.html

Q: Where can I find the CLI tools to manage EC2?

A: The AWS Command Line Interface, available under the DFSG-compliant Apache 2 license, can be installed via apt install awscli on Jessie systems and above. Historical note: the original Amazon EC2 API Tools were not DFSG-compliant, but Debian (still) distributes alternate set of DFSG-compliant tools, that are designed to be fully compatible, called euca2ools.

Q: Where can I find the list of Official Debian AMIs?

A: The following page has a list of Official and unofficial Debian AMIs: Cloud/AmazonEC2Image. See also 694035 for work in progress on a machine-readable list.

Q: How can I build my own AMI?

A: Stretch (and later) AMIs are created using the FAI tool using the debian-cloud-images configuration. An introduction into creating customized AMIs based on the FAI configuration can be found on Noah’s blog.

Packer is another popular tool for creating AMIs. It has the ability to integrate with existing configuration management systems such as chef and puppet, and be used to create images based on customizations performed on a running instance.

Anders Ingemann has created a build script for bootstrapping instances, and was used to create the official AMIs for jessie and earlier. The script can be automated as it needs no user interaction. Custom scripts can be attached to the process as well. You can download or clone the script from github. Any bugs or suggestions should be reported via the github issue tracker. The script is packaged and will be available for install starting with Debian Wheezy.

Also refer to Amazon’s documentation on this topic.

AWS EC2 at a glance

Posted on by
  1. Elastic Compute Cloud EC2 instance is like a remote computer running Windows or Linux and on which you can install whatever software you want, including a Web server running PHP code and a database server.
  2. EC2 is an Infrastructure as a Service (IaaS) Cloud Computing Platform provided by Amazon Web Services, that allows users to instantiate various types of virtual machines.
  3. EC2 provides scalable computing capacity in the Amazon Web Services (AWS) cloud. Using Amazon EC2 eliminates your need to invest in hardware up front, so you can develop and deploy applications faster.

Note: Don’t confuse EC2 with S3 because S3 is
is a repository for Internet data which provides access to reliable, fast, and inexpensive data storage infrastructure. S3 is designed to make web-scale computing easy by enabling you to store and retrieve any amount of data, at any time, from within Amazon EC2  OR anywhere on the web.

Login and click on ==> Resources

Login and click on ‘Resouce Groups’

Resources ==> EC2 (Under the Compute).

Services ==> EC2 ==> Running Instances

  1. You can launch new instance or connect by clicking the connect (next to ‘Launch Instance’).

2. Here are the different options under the ‘Actions’



Here you can check the ‘Description’, ‘Status Check’, ‘Monitoring’ ,’Tag’, and Usage Instructions’.

FAQs on Amazon Elastic Compute Cloud (EC2).